The present application relates to object detection and localized extremity guidance.
Recently, smart watches have been a very active area of research and development and various companies have released capable wrist-worn computers. For the blind, these wearable smart devices can be used to communicate event-based knowledge. For example, a watch can vibrate on the hour to indicate the passage of time, or vibrate in response to an incoming phone call instead of ringing. Vibration around the wrist is generally un-intrusive, but still informative.
The type of vibration used to alert a user can also be varied to convey different information. In U.S. Pre-grant publication 2006/0129308 A1 (US308), a system is described where different vibrations convey limited information about different types of objects detected in the environment using RFID tag-readers, such as identification codes identifying the objects. This information is relayed to a computer which makes use of it, such as alerting the user to the existence of a dangerous condition (e.g., a fire) using vibration. However, while US308 discusses the possibility of using the information from the RFID modules to ascertain the direction of travel, speed, and path of the user, US308 does not disclose any particular methods for computing the speed and path of the user, or using the path computation to guide an extremity of a user to and/or around object(s). Rather, US308 is limited to generally describing using RFID tags to define a grid, which is used to track a user's general movements.
Some tactile belt systems employ a haptic interface around the waist of a user for communicating directions in new environments and thus guiding people, such as those who are visually impaired, along arbitrarily complex paths. These systems are designed to work with robot(s) as a guide, which can detect obstacles or potentially other locations of interest using existing techniques, and guide the user to designated areas. However, the directional feedback provided by such a system is not localized and thus lacks adequate directional definition in some cases. In addition, such a guide robot can be complex and require extensive setup, training, and maintenance.
A system described by J. Rempel, “Glasses That Alert Travelers to Objects Through Vibration? An Evaluation of iGlasses by RNIB and AmbuTech,” AFB AccessWorld Magazine, vol. 13, no. 9, Sep. 2012 (Rempel), uses glasses configured to alert the visually impaired about objects using vibration. These glasses detect objects that may be in the path of the user using ultrasound, and vibrate to indicate their proximity and left or right direction relative to the objects. However, the system described by Rempel is inadequate for localized guidance as it provides even less well-defined directional information than the above belt system.
Thus, the above-described systems are limited to providing coarse-grained navigational assistance which is unsuitable for localized guidance, such as guiding a hand to a particular target. Furthermore, they are not integrated with object detection to find specific objects of interest a user may want to grab, which is not a trivial task.
In a related area, some systems use vibrotactile feedback for human-robot interaction, such as leader-follower scenarios involving multiple robots as described by S. Scheggi, F. Chinello, and D. Prattichizzo, “Vibrotactile haptic feedback for human-robot interaction in leader-follower tasks,” in PETRA, Crete Island, Greece, 2012 (Scheggi). In particular, Scheggi demonstrates how bracelets equipped with three vibro motors worn by the human leader of a human robot team can be used to improve team cohesion. The robots track the human's path, velocity, and expected trajectory, and warn the human when his/her motion would make following impossible. However, the system described by Scheggi does not guide the human to a particular location using vibration, but rather constrains his/her motions based on robot feedback. In addition, the system is incapable of detecting objects and/or guiding a person to those objects, and is thus not suitable for localized guidance applications.
Various techniques also exist for teaching people new motor skills for use in sports training, dancing, fixing bad posture, or some forms of physical therapy, such as therapy provided after the occurrence of a stroke. Traditionally, a trainer would watch a given pupil and give the pupil feedback including spoken directions, visual demonstrations, and manually moving the pupil's limbs into the right location. But paying for such training or therapy is expensive and unattainable for many people. As a result, researchers have begun developing computerized haptic interfaces to guide a person's motion, and therefore increase the quality and consistency of the training, and the number of people who have access to it.
Currently, researchers are investigating which haptic interface is the most suitable for teaching motor skills. For instance, the system described in J. Lieberman and C. Breazeal, “TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning,” IEEE Transactions on Robotics, vol. 23, no. 5, pp. 919-926, October 2007, uses vibration motors placed about the wrist and upper arm of a given person to help that person achieve the desired positioning of his/her arm. The system uses a room-mounted computer vision system configured to track the person's arm relative to a given orientation and uses vibration to help position the arm along the appropriate axis. In this way, a person's arm could be pushed into the right location using vibration.
Another system described by F. Sergi, D. Accoto, D. Compolo, and E. Guglielmelli, “Forearm orientation guidance with a vibrotactile feedback bracelet: On the directionality of tactile motor communication,” in Proc. of the Int. Conf. on Biomedical Robotics and Biomechatronics, Scottsdale, Ariz., 2008, pp. 133-138, uses vibrotactile feedback with a single bracelet to guide an arm along a trajectory, and explores various configurations for such a bracelet. Other implementations, such as those described by A. L. Guinan, N. C. Hornbaker, M. N. Montandon, A. J. Doxon, and W. Provancher, “Back-to-back skin stretch feedback for communicating five degree-of-freedom direction cues,” in IEEE World Haptics Conference, Daejeon, Korea, 2013 and K. Bark, J. Wheeler, P. Shull, J. Savall, and M. Cutkosky, “Rotational Skin Stretch Feedback: A Wearable Haptic Display for Motion,” IEEE Transactions on Haptics, vol. 3, no. 3, pp. 166-176, July 2010, use skin stretch instead of vibration as another modality for guiding hands into a desired rotational form. The system described by M. F. Rotella, K. Guerin, Xingchi He, and A. M. Okamura, “HAPI Bands: A haptic augmented posture interface,” in HAPTICS Symposium, Vancouver, BC, 2012 uses a set of five bands placed around a person's wrists, elbows and waist, which guide the person to a correct posture (e.g., Yoga) in response to a computer vision-based analysis. This system uses a Kinect™ camera to estimate the person's body skeleton and generate vibrational error.
While some of above haptic systems may be designed to use vibratory feedback to achieve particular postures and/or actions, these systems lack the capability to detect objects in the environment and provide localized navigational assistance to the user to reach out and manipulate detected objects.